In [7]:
%matplotlib inline
import seaborn as sns
sns.heatmap(hm[::-1], cmap='viridis')
Out[7]:
<AxesSubplot:xlabel='n2', ylabel='n1'>
This tutorial will show how to optimize strategies with multiple parameters and how to examine and reason about optimization results. It is assumed you're already familiar with basic backtesting.py usage.
First, let's again import our helper moving average function. In practice, one should use functions from an indicator library, such as TA-Lib or Tulipy.
Our strategy will be a similar moving average cross-over strategy to the one in Quick Start User Guide, but we will use four moving averages in total: two moving averages whose relationship determines a general trend (we only trade long when the shorter MA is above the longer one, and vice versa), and two moving averages whose cross-over with daily close prices determine the signal to enter or exit the position.
from backtesting import Strategy
from backtesting.lib import crossover
class Sma4Cross(Strategy):
n1 = 50
n2 = 100
n_enter = 20
n_exit = 10
def init(self):
self.sma1 = self.I(SMA, self.data.Close, self.n1)
self.sma2 = self.I(SMA, self.data.Close, self.n2)
self.sma_enter = self.I(SMA, self.data.Close, self.n_enter)
self.sma_exit = self.I(SMA, self.data.Close, self.n_exit)
def next(self):
if not self.position:
# On upwards trend, if price closes above
# "entry" MA, go long
# Here, even though the operands are arrays, this
# works by implicitly comparing the two last values
if self.sma1 > self.sma2:
if crossover(self.data.Close, self.sma_enter):
self.buy()
# On downwards trend, if price closes below
# "entry" MA, go short
else:
if crossover(self.sma_enter, self.data.Close):
self.sell()
# But if we already hold a position and the price
# closes back below (above) "exit" MA, close the position
else:
if (self.position.is_long and
crossover(self.sma_exit, self.data.Close)
or
self.position.is_short and
crossover(self.data.Close, self.sma_exit)):
self.position.close()
It's not a robust strategy, but we can optimize it.
Grid search is an exhaustive search through a set of specified sets of values of hyperparameters. One evaluates the performance for each set of parameters and finally selects the combination that performs best.
Let's optimize our strategy on Google stock data using randomized grid search over the parameter space, evaluating at most (approximately) 200 randomly chosen combinations:
%%time
from backtesting import Backtest
from backtesting.test import GOOG
backtest = Backtest(GOOG, Sma4Cross, commission=.002)
stats, heatmap = backtest.optimize(
n1=range(10, 110, 10),
n2=range(20, 210, 20),
n_enter=range(15, 35, 5),
n_exit=range(10, 25, 5),
constraint=lambda p: p.n_exit < p.n_enter < p.n1 < p.n2,
maximize='Equity Final [$]',
max_tries=200,
random_state=0,
return_heatmap=True)
CPU times: user 194 ms, sys: 7.79 ms, total: 202 ms Wall time: 7.56 s
Notice return_heatmap=True parameter passed to
Backtest.optimize().
It makes the function return a heatmap series along with the usual stats of the best run.
heatmap is a pandas Series indexed with a MultiIndex, a cartesian product of all permissible (tried) parameter values.
The series values are from the maximize= argument we provided.
heatmap
n1 n2 n_enter n_exit
20 60 15 10 10102.87
80 15 10 9864.22
100 15 10 11003.22
30 40 20 15 11771.29
25 15 16178.55
...
100 200 15 10 13118.25
20 10 11308.46
15 16350.94
25 10 8991.55
30 10 9953.07
Name: Equity Final [$], Length: 177, dtype: float64
This heatmap contains the results of all the runs, making it very easy to obtain parameter combinations for e.g. three best runs:
heatmap.sort_values().iloc[-3:]
n1 n2 n_enter n_exit
100 120 15 10 18159.06
160 20 15 19216.54
50 160 20 15 19565.69
Name: Equity Final [$], dtype: float64
But we use vision to make judgements on larger data sets much faster.
Let's plot the whole heatmap by projecting it on two chosen dimensions.
Say we're mostly interested in how parameters n1 and n2, on average, affect the outcome.
hm = heatmap.groupby(['n1', 'n2']).mean().unstack()
hm
| n2 | 40 | 60 | 80 | 100 | 120 | 140 | 160 | 180 | 200 |
|---|---|---|---|---|---|---|---|---|---|
| n1 | |||||||||
| 20 | NaN | 10102.87 | 9864.22 | 11003.22 | NaN | NaN | NaN | NaN | NaN |
| 30 | 13974.92 | 11696.32 | 11757.99 | 15092.99 | 13152.24 | 11518.69 | 11271.35 | 11384.55 | 10649.05 |
| 40 | NaN | 13666.45 | NaN | 7549.10 | 10629.48 | 12860.99 | 11405.29 | 10863.81 | 10658.14 |
| 50 | NaN | 8383.46 | 10180.50 | 10563.79 | 9081.95 | 14272.27 | 13575.86 | 11383.46 | 10053.47 |
| 60 | NaN | NaN | 9232.42 | 8046.49 | 10838.45 | 12876.59 | 10312.95 | 9427.55 | 9555.40 |
| 70 | NaN | NaN | 14712.14 | 7192.89 | 10403.01 | 10065.28 | 8293.73 | 9895.78 | 9360.48 |
| 80 | NaN | NaN | NaN | 10863.11 | 7721.24 | 9139.95 | 8813.95 | 10414.66 | 8908.49 |
| 90 | NaN | NaN | NaN | 8958.14 | 9538.05 | 9884.42 | 9685.92 | 11343.64 | 8806.57 |
| 100 | NaN | NaN | NaN | NaN | 11253.16 | 7101.26 | 11323.43 | 10163.32 | 11944.46 |
Let's plot this table using the excellent Seaborn package:
%matplotlib inline
import seaborn as sns
sns.heatmap(hm[::-1], cmap='viridis')
<AxesSubplot:xlabel='n2', ylabel='n1'>
We see that, on average, we obtain the highest result using trend-determining parameters n1=40 and n2=60,
and it's not like other nearby combinations work similarly well — in our particular strategy, this combination really stands out.
Since our strategy contains several parameters, we might be interested in other relationships between their values.
We can use
backtesting.lib.plot_heatmaps()
function to plot interactive heatmaps of all parameter combinations simultaneously.
from backtesting.lib import plot_heatmaps
plot_heatmaps(heatmap, agg='mean')
Above, we used randomized grid search optimization method. Any kind of grid search, however, might be computationally expensive for large data sets. In the follwing example, we will use scikit-optimize package to guide our optimization better informed using forests of decision trees. The hyperparameter model is sequentially improved by evaluating the expensive function (the backtest) at the next best point, thereby hopefully converging to a set of optimal parameters with as few evaluations as possible.
So, with method="skopt":
%%capture
! pip install scikit-optimize # This is a run-time dependency
%%time
stats_skopt, heatmap, optimize_result = backtest.optimize(
n1=[10, 100], # Note: For method="skopt", we
n2=[20, 200], # only need interval end-points
n_enter=[10, 40],
n_exit=[10, 30],
constraint=lambda p: p.n_exit < p.n_enter < p.n1 < p.n2,
maximize='Equity Final [$]',
method='skopt',
max_tries=200,
random_state=0,
return_heatmap=True,
return_optimization=True)
CPU times: user 20.2 s, sys: 84.3 ms, total: 20.3 s Wall time: 20.3 s
heatmap.sort_values().iloc[-3:]
n1 n2 n_enter n_exit 35 98 28 24 28365.15 68 96 29 24 28424.02 44 134 39 27 29941.38 Name: Equity Final [$], dtype: float64
Notice how the optimization runs somewhat slower even though max_tries= is the same. But that's due to the sequential nature of the algorithm and should actually perform rather comparably even in cases of much larger parameter spaces where grid search would effectively blow up, but likely (hopefully) reaching a better local optimum than a randomized search would.
A note of warning, again, to take steps to avoid
overfitting
insofar as possible.
Understanding the impact of each parameter on the computed objective function is easy in two dimensions, but as the number of dimensions grows, partial dependency plots are increasingly useful. Plotting tools from scikit-optimize take care of many of the more mundane things needed to make good and informative plots of the parameter space:
from skopt.plots import plot_objective
_ = plot_objective(optimize_result, n_points=10)
from skopt.plots import plot_evaluations
_ = plot_evaluations(optimize_result, bins=10)
Learn more by exploring further examples or find more framework options in the full API reference.